Use Of Deep Reinforcement Learning Research Articles

Exploring the Use of Deep Reinforcement Learning for Autonomous Navigation in Unstructured Environments

Abstract: Autonomous navigation in unstructured environments poses a significant challenge in robotics and artificial intelligence. The capability to navigate through dynamic and unpredictable terrains such as disaster zones, outdoor landscapes, or congested urban settings demands sophisticated solutions. This research paper delves into the imperative role of Deep Reinforcement Learning (DRL) in addressing these challenges and advancing the field of autonomous navigation. The core necessity of this research paper lies in the application and exploration of DRL within the realm of autonomous navigation. By leveraging neural networks and reinforcement learning algorithms, autonomous agents can dynamically navigate through unstructured environments without explicit programming or human intervention. Instead, they learn to navigate by receiving feedback through rewards or penalties, thereby continuously improving their decision-making processes. Through a comprehensive review of existing literature and experiments this paper aims to elucidate the pivotal role of DRL in shaping the future of autonomous navigation. It highlights the necessity of robust and adaptive systems capable of navigating unstructured environments, emphasizing the transformative potential of DRL in revolutionizing autonomous systems' capabilities.

Learning Scene Representations for Human-assistive Displays Using Self-attention Networks

Video-see-through (VST) augmented reality (AR) is widely used to present novel augmentative visual experiences by processing video frames for viewers. Among VST AR systems, assistive vision displays aim to compensate for low vision or blindness, presenting enhanced visual information to support activities of daily living for the vision impaired/deprived. Despite progress, current assistive displays suffer from a visual information bottleneck, limiting their functional outcomes compared to healthy vision. This motivates the exploration of methods to selectively enhance and augment salient visual information. Traditionally, vision processing pipelines for assistive displays rely on hand-crafted, single-modality filters, lacking adaptability to time-varying and environment-dependent needs. This article proposes the use of Deep Reinforcement Learning (DRL) and Self-attention (SA) networks as a means to learn vision processing pipelines for assistive displays. SA networks selectively attend to task-relevant features, offering a more parameter—and compute-efficient approach to RL-based task learning. We assess the feasibility of using SA networks in a simulation-trained model to generate relevant representations of real-world states for navigation with prosthetic vision displays. We explore two prosthetic vision applications, vision-to-auditory encoding, and retinal prostheses, using simulated phosphene visualisations. This article introduces SA-px, a general-purpose vision processing pipeline using self-attention networks, and SA-phos, a display-specific formulation targeting low-resolution assistive displays. We present novel scene visualisations derived from SA image patches importance rankings to support mobility with prosthetic vision devices. To the best of our knowledge, this is the first application of self-attention networks to the task of learning vision processing pipelines for prosthetic vision or assistive displays.

Beacon, a Lightweight Deep Reinforcement Learning Benchmark Library for Flow Control

Recently, the increasing use of deep reinforcement learning for flow control problems has led to a new area of research focused on the coupling and adaptation of the existing algorithms to the control of numerical fluid dynamics environments. Although still in its infancy, the field has seen multiple successes in a short time span, and its fast development pace is certainly partly imparted by the open-source effort that drives the expansion of the community. Yet this emerging domain is still missing a common ground to (i) ensure the reproducibility of the results and (ii) offer a proper ad hoc benchmarking basis. To this end, we propose beacon, an open-source benchmark library composed of seven lightweight one-dimensional and two-dimensional flow control problems with various characteristics, action and observation space characteristics, and CPU requirements. In this contribution, the seven considered problems are described, and reference control solutions are provided. The sources for the following work are publicly available.

Artificial intelligence-based position control: reinforcement learning approach in spring mass damper systems

This work examines the use of deep Reinforcement Learning (RL) in mass-spring system position control, providing a fresh viewpoint that goes beyond conventional control techniques. Mass-spring systems are widely used in many sectors and are basic models in control theory. The novel aspect of this approach is the thorough examination of the impact of several optimizer algorithms on the RL methodology, which reveals the optimal control tactics. The research applies a Deep Deterministic Policy Gradient (DDPG) algorithm for continuous action spaces, where the actor and critic networks are important components in assessing the agent’s performance. The RL agent is trained to follow a reference trajectory using the Simulink environment for system modeling. The study provides insights into the agent’s learning approach and performance optimization by evaluating the training process using force-time graphs, reward graphs, and Episode Manager charts. Furthermore, the effect of different combinations of optimizers on the control performance of the agent is examined. The outcomes highlight the importance of optimizer selection in the learning process by revealing significant variations in training times. As a result, a better understanding of the relationship between various optimizers and control performance is provided by this study’s novel application of reinforcement learning in mass-spring system control. The results raise the possibility of more potent methods for controlling complex systems and add to the expanding field of study at the interface of control theory and deep learning.

Cooperative Coverage Path Planning for Multi-Mobile Robots Based on Improved K-Means Clustering and Deep Reinforcement Learning

With the increasing complexity of patrol tasks, the use of deep reinforcement learning for collaborative coverage path planning (CPP) of multi-mobile robots has become a new hotspot. Taking into account the complexity of environmental factors and operational limitations, such as terrain obstacles and the scope of the task area, in order to complete the CPP task better, this paper proposes an improved K-Means clustering algorithm to divide the multi-robot task area. The improved K-Means clustering algorithm improves the selection of the first initial clustering point, which makes the clustering process more reasonable and helps to distribute tasks more evenly. Simultaneously, it introduces deep reinforcement learning with a dueling network structure to better deal with terrain obstacles and improves the reward function to guide the coverage process. The simulation experiments have confirmed the advantages of this method in terms of balanced task assignment, improvement in strategy quality, and enhancement of coverage efficiency. It can reduce path duplication and omission while ensuring coverage quality.

Deep reinforcement learning for base station switching scheme with federated LSTM‐based traffic predictions

AbstractTo meet increasing traffic requirements in mobile networks, small base stations (SBSs) are densely deployed, overlapping existing network architecture and increasing system capacity. However, densely deployed SBSs increase energy consumption and interference. Although these problems already exist because of densely deployed SBSs, even more SBSs are needed to meet increasing traffic demands. Hence, base station (BS) switching operations have been used to minimize energy consumption while guaranteeing quality‐of‐service (QoS) for users. In this study, to optimize energy efficiency, we propose the use of deep reinforcement learning (DRL) to create a BS switching operation strategy with a traffic prediction model. First, a federated long short‐term memory (LSTM) model is introduced to predict user traffic demands from user trajectory information. Next, the DRL‐based BS switching operation scheme determines the switching operations for the SBSs using the predicted traffic demand. Experimental results confirm that the proposed scheme outperforms existing approaches in terms of energy efficiency, signal‐to‐interference noise ratio, handover metrics, and prediction performance.

Cost-aware cloud workflow scheduling using DRL and simulated annealing

Cloud workloads are highly dynamic and complex, making task scheduling in cloud computing a challenging problem. While several scheduling algorithms have been proposed in recent years, they are mainly designed to handle batch tasks and not well-suited for real-time workloads. To address this issue, researchers have started exploring the use of Deep Reinforcement Learning (DRL). However, the existing models are limited in handling independent tasks and cannot process workflows, which are prevalent in cloud computing and consist of related subtasks. In this paper, we propose SA-DQN, a scheduling approach specifically designed for real-time cloud workflows. Our approach seamlessly integrates the Simulated Annealing (SA) algorithm and Deep Q-Network (DQN) algorithm. The SA algorithm is employed to determine an optimal execution order of subtasks in a cloud server, serving as a crucial feature of the task for the neural network to learn. We provide a detailed design of our approach and show that SA-DQN outperforms existing algorithms in terms of handling real-time cloud workflows through experimental results.

Deep reinforcement learning‐based resource allocation in multi‐access edge computing

SummaryNetwork architects and engineers face challenges in meeting the increasing complexity and low‐latency requirements of various services. To tackle these challenges, multi‐access edge computing (MEC) has emerged as a solution, bringing computation and storage resources closer to the network's edge. This proximity enables low‐latency data access, reduces network congestion, and improves quality of service. Effective resource allocation is crucial for leveraging MEC capabilities and overcoming limitations. However, traditional approaches lack intelligence and adaptability. This study explores the use of deep reinforcement learning (DRL) as a technique to enhance resource allocation in MEC. DRL has gained significant attention due to its ability to adapt to changing network conditions and handle complex and dynamic environments more effectively than traditional methods. The study presents the results of applying DRL for efficient and dynamic resource allocation in MEC Computing, optimizing allocation decisions based on real‐time environment and user demands. By providing an overview of the current research on resource allocation in MEC using DRL, including components, algorithms, and the performance metrics of various DRL‐based schemes, this review article demonstrates the superiority of DRL‐based resource allocation schemes over traditional methods in diverse MEC conditions. The findings highlight the potential of DRL‐based approaches in addressing challenges associated with resource allocation in MEC.

Explainability in Deep Reinforcement Learning: A Review into Current Methods and Applications

The use of Deep Reinforcement Learning (DRL) schemes has increased dramatically since their first introduction in 2015. Though uses in many different applications are being found, they still have a problem with the lack of interpretability. This has bread a lack of understanding and trust in the use of DRL solutions from researchers and the general public. To solve this problem, the field of Explainable Artificial Intelligence has emerged. This entails a variety of different methods that look to open the DRL black boxes, ranging from the use of interpretable symbolic Decision Trees to numerical methods like Shapley Values. This review looks at which methods are being used and for which applications. This is done to identify which models are the best suited to each application or if a method is being underutilised.

Optimizing Automatic Voltage Regulation: A Deep Reinforcement Learning Approach

This paper presents a novel approach to enhance the performance of Automatic Voltage Regulator (AVR) systems in power systems using Deep Reinforcement Learning (DRL). The AVR plays a critical role in maintaining voltage stability and ensuring reliable power delivery. However, conventional control strategies, such as PID controllers, have limitations in handling complex and nonlinear power system dynamics. In this study, the application of DRL techniques is explored, particularly the Twin-Delayed Deep Deterministic Policy Gradient (TD3) algorithm, to AVR control. This algorithm offers the advantage of handling continuous action spaces and enable the controller to learn optimal control policies directly from the system's state information. The results show that the DRL approach outperforms the traditional PID and Neural Network-based control approaches, with the shortest time response and the best voltage regulation performance. The use of DRL in AVR system control shows promising potential for improving the efficiency and accuracy of power system control. This research provides insights into the advantages of DRL for process control and highlights its potential for future applications in power system control.

Flexible operation and maintenance optimization of aging cyber-physical energy systems by deep reinforcement learning

Cyber-Physical Energy Systems (CPESs) integrate cyber and hardware components to ensure a reliable and safe physical power production and supply. Renewable Energy Sources (RESs) add uncertainty to energy demand that can be dealt with flexible operation (e.g., load-following) of CPES; at the same time, scenarios that could result in severe consequences due to both component stochastic failures and aging of the cyber system of CPES (commonly overlooked) must be accounted for Operation & Maintenance (O&M) planning. In this paper, we make use of Deep Reinforcement Learning (DRL) to search for the optimal O&M strategy that, not only considers the actual system hardware components health conditions and their Remaining Useful Life (RUL), but also the possible accident scenarios caused by the failures and the aging of the hardware and the cyber components, respectively. The novelty of the work lies in embedding the cyber aging model into the CPES model of production planning and failure process; this model is used to help the RL agent, trained with Proximal Policy Optimization (PPO) and Imitation Learning (IL), finding the proper rejuvenation timing for the cyber system accounting for the uncertainty of the cyber system aging process. An application is provided, with regards to the Advanced Lead-cooled Fast Reactor European Demonstrator (ALFRED).

Fully dynamic reorder policies with deep reinforcement learning for multi-echelon inventory management

The operation of inventory systems plays an important role in the success of manufacturing companies, making it a highly relevant domain for optimization. In particular, the domain lends itself to being approached via Deep Reinforcement Learning (DRL) models due to it requiring sequential reorder decisions based on uncertainty to minimize cost. In this paper, we evaluate state-of-the-art optimization approaches to determine whether Deep Reinforcement Learning can be applied to the multi-echelon inventory optimization (MEIO) framework in a practically feasible manner to generate fully dynamic reorder policies. We investigate how it performs in comparison to an optimized static reorder policy, how robust it is when it comes to structural changes in the environment, and whether the use of DRL is safe in terms of risk in real-world applications. Our results show promising performance for DRL with potential for improvement in terms of minimizing risky behavior.

A hybrid heuristic-reinforcement learning-based real-time control model for residential behind-the-meter PV-battery systems

The behind-the-meter (BTM) energy management problem has recently attracted a lot of attention due to the increase in the number of residential photovoltaic (PV)-battery energy storage systems (BESSs). In this work, the use of deep reinforcement learning (DRL) combined with a novel heuristic model for real-time control of home batteries is investigated. The control problem is formulated as a finite Markov Decision Process with discrete time steps, where a proximal policy optimization (PPO) algorithm is employed to train the DRL agent with discrete action space. The agent is trained using real-world measured data to learn the policy for sequential charge/discharge tasks, aiming to minimize daily electricity costs. The battery power is calculated using an innovative heuristic model considering the agent's decision and the battery's available capacity, ensuring demand-supply balance through PV self-consumption and load demand shifting. The performance of the model is evaluated by comparing it to four RL agents and two benchmark models based on rule-based and scenario-based stochastic optimization strategies. The results confirm that the presented model outperforms its counterparts, offering €80.38 savings on electricity bills over 46 days of the test data set. This figure exceeds the savings of the rule-based and stochastic models by €15.64 and €19.38, respectively.

The Application of Deep Reinforcement Learning in Stock Trading Models

Deep Reinforcement Learning (DRL), which integrates the perceptual strength of Deep Learning (DL) with the determination strength of Reinforcement Learning (RL), has emerged as an advanced approach in stock trading. This article focuses on summarizing the research on DRL in stock trading over the past five years, with an emphasis on state definition, action design, reward design, and algorithm selection in stock trading models. Many studies have found that the use of DRL in stock trading can effectively improve investment returns and profitability. In the last years, the adoption of DRL in the stock market has increased and researchers have achieved higher returns and substantial profits through continuous model optimization. However, current research on DRL models faces challenges because of the need for large amounts of complex and uncertain stock market data and the impacts of market volatility and the influence of information asymmetry. This review compares the discrepancies in processing logic among various studies and summarizes the progress made in existing research. It also explores the current challenges and limitations, and discusses potential improvement directions for DRL models in stock trading.

A Hierarchical Deep Learning Approach for Optimizing CCA Threshold and Transmit Power in Wi-Fi Networks

The nodes, e.g., access points and clients, in current WiFi networks rely on carrier sense multiple access (CSMA) for channel access. This means they rely on a clear channel assessment (CCA) threshold to determine when it is safe to transmit a packet. In practice, this threshold is fixed, and nodes do not adapt their CCA threshold given varying channel conditions. To this end, this paper proposes and studies the use of deep reinforcement learning (DRL) to optimize the CCA threshold and transmit power of an AP. It allows an AP to learn the optimal policy for setting its CCA threshold and transmit power given its queue length and interference level. We outline a semi-Markov decision process (Semi-MDP) for the problem at hand and solve it using a decentralized deep hierarchical reinforcement learning (HRL) approach. We then evaluate its performance using traffic generated via a Poisson distribution and also from a trace file. The results show that when compared to two competing solutions, an AP with HRL reduces its average queue length by 24.4% and 52.4%, respectively.

DEEP REINFORCEMENT LEARNING ON STOCK DATA

This study proposes using Deep Reinforcement Learning (DRL) for stock trading decisions and prediction. DRL is a machine learning technique that enables agents to learn optimal strategies by interacting with their environment. The proposed model surpasses traditional models and can make informed trading decisions in real-time. The study highlights the feasibility of applying DRL in financial markets and its advantages in strategic decision- making. The model's ability to learn from market dynamics makes it a promising approach for stock market forecasting. Overall, this paper provides valuable insights into the use of DRL for stock trading decisions and prediction, establishing a strong case for its adoption in financial markets. Keywords: reinforcement learning, stock market, deep reinforcement learning.

Practical Application of Deep Reinforcement Learning to Optimal Trade Execution

Although deep reinforcement learning (DRL) has recently emerged as a promising technique for optimal trade execution, two problems still remain unsolved: (1) the lack of a generalized model for a large collection of stocks and execution time horizons; and (2) the inability to accurately train algorithms due to the discrepancy between the simulation environment and real market. In this article, we address the two issues by utilizing a widely used reinforcement learning (RL) algorithm called proximal policy optimization (PPO) with a long short-term memory (LSTM) network and by building our proprietary order execution simulation environment based on historical level 3 market data of the Korea Stock Exchange (KRX). This paper, to the best of our knowledge, is the first to achieve generalization across 50 stocks and across an execution time horizon ranging from 165 to 380 min along with dynamic target volume. The experimental results demonstrate that the proposed algorithm outperforms the popular benchmark, the volume-weighted average price (VWAP), highlighting the potential use of DRL for optimal trade execution in real-world financial markets. Furthermore, our algorithm is the first commercialized DRL-based optimal trade execution algorithm in the South Korea stock market.

Reinforcement and Curriculum Learning for Off-Road Navigation of an UGV with a 3D LiDAR

This paper presents the use of deep Reinforcement Learning (RL) for autonomous navigation of an Unmanned Ground Vehicle (UGV) with an onboard three-dimensional (3D) Light Detection and Ranging (LiDAR) sensor in off-road environments. For training, both the robotic simulator Gazebo and the Curriculum Learning paradigm are applied. Furthermore, an Actor-Critic Neural Network (NN) scheme is chosen with a suitable state and a custom reward function. To employ the 3D LiDAR data as part of the input state of the NNs, a virtual two-dimensional (2D) traversability scanner is developed. The resulting Actor NN has been successfully tested in both real and simulated experiments and favorably compared with a previous reactive navigation approach on the same UGV.

Using Deep Reinforcement Learning for Assessing the Consequences of Cyber Mitigation Techniques on Industrial Control Systems

This paper discusses an in-progress study involving the use of deep reinforcement learning (DRL) to mitigate the effects of an advanced cyber-attack against industrial control systems (ICS).  The research is a qualitative, exploratory study which emerged as a gap during the execution of two rapid prototyping studies.  During these studies, cyber defensive procedures, known as “Mitigation, were characterized as actions taken to minimize the impact of ongoing advanced cyber-attacks against an ICS while enabling primary operations to continue.  To execute Mitigation procedures, affected ICS components required rapid isolation and quarantining from “healthy” system segments. However today, with most attacks leveraging automation, mitigation also requires rapid decision-making capabilities operating at the speed of automation yet with human-like refinement.  The authors settled on the choice of DRL as a viable solution to this problem due to the algorithm’s designs which involves “intelligent” decisions based upon continuous learning achieved through a rewards system.  The primary theory of this study posits that processes informed by data sources relative to the execution path of an advanced cyber-attack as well as the consequences of deploying a particular Mitigation procedure evolve the system into an ever-improving defensive capability.  This study seeks to produce a defensive DLR based software agent trained by a DRL based offensive software agent that generates policy refinements based upon extrapolations from a corrupted network state as reported by an IDS and baseline data. Results include an estimation rule that would quantify impacts of various mitigation actions while protecting the operational critical path and isolating an in-progress attack.  This study is in a conceptual phase and development has not started.
 This research questions for this study are:
 RQ1: Can this software agent categorize correctly an in-progress cyber-attack and extrapolate the potential ICS assets affected?
 RQ2: Can this software agent categorize novel cyber-attacks and extrapolate a probable attack vector while enumerating affected assets?
 RQ3: Can this software agent characterize how operations are affected by quarantine actions?
 RQ4: Can this software agent generate a set of ranked recommended courses of action by effectiveness, and least negative effects on the operational critical path?

Graph Transformer with Reinforcement Learning for Vehicle Routing Problem

Vehicle routing problem (VRP) is one of the classic combinatorial optimization problems where an optimal tour to visit customers is required with a minimum total cost in the presence of some constraints. Recently, VRP is being solved with the use of deep reinforcement learning (DRL), with node sets considered (represented) as a graph structure. Existing Transformer based DRL solutions for VRP rely only on node information ignoring the role of information on the edges between nodes in the graph structure. In this paper, we proposed an attention‐based end‐to‐end DRL model to solve VRP which embeds edge information between nodes for rich graph representation learning. We used Transformer based encoder‐decoder framework with an edge information embedded multi‐head attention (EEMHA) layer in the encoder. The EEMHA‐based encoder learns underlying structure of the graph and generates an expressive graph topology representation by merging node and edge information. We trained our model using proximal policy optimization (PPO) with some code‐level optimization techniques. We conducted an experiment on randomly generated instances and on a real‐world data generated from road networks to verify the performance of our proposed model. The result from all experiments show that our model performs better than the existing DRL methods and most of the conventional heuristics with good generalizability from random instance training to real‐world instance testing of different problem size. © 2023 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.